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SPECIFICATION 

TITLE OF THE INVENTION 

SURGICAL OPERATION ASSISTANCE SYSTEM AND SURGICAL OPERATION 

ASSISTING METHOD 

5 BACKGROUND OF THE INVENTION 

The present invention relates to an operation assistance 
system, and in particular, it relates to a surgical operation 
assisting system, being suitable for use of assistance of an 
orthopedic surgical operation, with using a computer therein. 

10 In recent years, accompanying with the development of image 

devices for picking up an image for medical use, a method is widely 
spread for planning a surgical operation by referring to an image 
that is picked up for the medical use (herein after, a "medical 
use image"), which is displayed on a computer. Accompanying with 

15 this, an apparatus is also developed for assisting the surgical 
operation plan on the computer. 

An example of such the conventional technology is described 
in, such as, Japanese Patent Laying-Open No. 2000-113086 (2000), 
Japanese Patent Laying-Open No. 2001-293007 (2001), and Japanese 

20 Patent Laying-Open No. Hei 11-155881 (1999), for example. The 
surgical operation assistance system described in those 
publications , comprises : an image picking-up apparatus for picking 
up a surgical filed; an image producing portion for producing a 
stereoscopy (i.e. , three dimensional vision) of the surgical field, 

25 which is picked up; a surgical operation route calculating portion 
for calculating out a route for the surgical operation; and an 
image display apparatus for displaying the stereoscopy and the 




surgical operation route thereon. 

Such the operation assistance system described in each of 
those publications mentioned above is used, however mainly, in 
cerebral surgical operation, for the purpose of obtaining the 
5 operation route of low invasion, up to the surgical field, with 
using themedical use image of a patient, which is picked up . However, 
no consideration is taken into consideration about a possibility 
of using such the system in the orthopedic surgical operation of 
the surgical field, such as, the osteotomy, for example. In 
10 particular, in the orthopedic surgical operation, it is necessary 
to determine the smooth operation route relating to the surgical 
field, however the surgical operation assistance systems described 
in those publications mentioned above cannot cope with such. 

BRIEF SUMMARY OF THE INVENTION 

15 An object is, according to the present invention, to provide 

a surgical operation assistance system and a surgical operation 
assisting method, and also a program thereof, with which such the 
smooth operation route can be determined easily, up to the surgical 
field, and thereby enabling the orthopedic surgical operation, 

20 with ease and certainty. 

For accomplishing the object mentioned above, according to 
the present invention, there is provided a surgical operation 
assistance system, comprising; an image pick-up apparatus for 
picking up an image of a surgical filed; an image producing unit 

25 for producing a stereographic image of the surgical filed, the 
image of which is picked up; an input unit for inputting reference 
points of a surgical operation route upon basis of a kind of the 
surgical operation and said stereographic image; a surgical 
operation route calculation unit for calculating a smooth surgical 

30 operation route upon basis of said kind of the surgical operation 
inputted and the reference points; an image processing unit for 
processing said stereographic image and said surgical operation 
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route to be displayable; and an image displaying apparatus for 
displaying the images processed in said image processing unit 
thereon. 

Herein, according to the present invention, more preferably, 
5 the structure of the surgical operation assistance system as 
described in the above are as follows. Thus, it further comprises 
an image extracting unit for extracting a partial image from said 
stereographic image, wherein said image processing unit processes 
the extracted image to be displayable. Also, it further comprises 

10 a slice image arbitrary line input unit for designating an arbitrary 
lineof a sliced image to be displayed, wherein said image processing 
unit processes the sliced image to be displayable, upon basis of 
the arbitrary line designated for the slice image. And, also, it 
further comprises a surgical operation robot for automatically 

15 conduct the surgical operation upon the surgical filed with using 
the surgical operation tool along with said surgical operation 
route calculated out. 

Also, for achieving the object mentioned above, according 
to the present invention, there is also provided a surgical 

20 operation assistance system, comprising: an image pick-up 
apparatus for picking up an image of a surgical filed; a surgical 
operation robot for conducting surgical operation upon the surgical 
field with using a surgical operation tool ; a position information 
integration unit for integrating position information of said 

25 surgical operation robot with an image of the surgical field picked 
up by said image pick-up apparatus; an image producing unit for 
producing a stereographic image of the surgical field picked up, 
and for producing an image piling up an image of said surgical 
operation tool on said stereographic image, upon basis of the 

30 information integrated in said position information integration 
unit; a reference point inputting unit for inputting reference 
points of a surgical operation route upon basis of a kind of the 
surgical operation tool and said stereographic image; a surgical 
operation route calculating unit for calculating out a smooth 
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surgrcal operation route upon basis of the kind of the surgical 
operate tool and the reference points, which are inputted 
rmage processing unit for processing said stereographic image'and 
sard surgrcal operation route to he displace under a desired 
condrtron; and an image displaying apparatus for displaying the 
rmage processed in said image processing unit thereon 

in th Hereln ' aCC ° rdingt0the P— "invention, more preferably, 
rn the surgrcal operation assistance system as described in the 
above, sard surgical operation robot is also operable upon the 
- surgrcal filed, manually, with using the surgloal * ™ 

along wrth the surgioal operatlon route caicuiated Md 

surgrcal operatron robot is exchangeable between the automatic 
operatron and the manual operation thereof. 

to th A " d ' aChieVi " 9 the ob ^ -entioned above, according 
to tbe present rnvention, there is further provided a surgical 
operatron assrstin, method, comprising the following step! of 
prcRrn, up an image of a surgical filed by means of an image pL-up 
apparatus; producing a stereographic image of the surgical filed 

reLTelT T * *" *"» »«= ' ^"ting 

LnTof th 3 SUr9lCal ° Peratl0n IOUte »-*■ of a 

*rnd of the surgrcal operation and said stereographic image through 

an r„ t rt; calculating a smooth surgical operation route upon 

re e en °* ****** - ^ 

reference pornts in a surgical operation route calculation unit- 

processrng said stereographic image and said surgical operation 
route to be displayable in an image processing unit 'and display ng 
the processed in said image processing unit on an L 9 l 

drsplayrng apparatus. 9 

And also, for achieving the object mentioned above, 
accordrn, to the present invention, there is further provided a 
surgrcal operation assisting method, comprising the following 
steps of, proving up an image of a surgical filed by means of an 

™ P ai C f"i U el; PParatUS; C ° ndUCti " 9 ™ al °<>™ ™ 
surgical field, manually, with n«<„„ a „ 

y ' Wltn usin 9 a surgical operation tool 
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of a surgical operation robot; integrating position information 
of said surgical operation robot with an image of the surgical 
field picked up by said image pick-up apparatus in a position 
information integration unit; producing a stereographic image of 
5 the surgical field picked up, and for producing an image piling 
up an image of said surgical operation tool on said stereographic 
image, upon basis of the information integrated in said position 
information integration unit , in an image producing unit ; inputting 
reference points of a surgical operation route upon basis of a 

10 kind of the surgical operation tool and said stereographic image 
through a reference point inputting unit; calculating out a smooth 
surgical operation route upon basis of the kind of the surgical 
operation tool and the reference points, which are inputted, in 
a surgical operation route calculating unit; processing said 

15 stereographic image and said surgical operation route to be 
displayable under a desired condition in an image processing unit; 
and displaying the image processed in said image processing unit 
on an image displaying apparatus. 

Herein, more preferably , according to the present invention, 
20 in the surgical operation assisting method as described in the 
above, the image of the surgical field is picked up by means of 
the image pick-up apparatus under a condition where markers are 
attached thereon in a number of three ( 3 ) or more , thereby producing 
a medical use image, further comprising the following steps of: 
25 attaching a same number of markers on an actual patient at positions 
where said makers are attached, measuring position coordinates 
of those markers through a three-dimension position measuring 

apparatus, thereby presenting them in a form of a matrix of 3x3 

or more; converting this matrix into a matrix of 3x3 or more for 
30 presenting the position coordinates of the markers on said medical 
use image in said position information integration unit; and 
producing an image piling up the image of said surgical operation 
tool on said stereographic image, upon said matrix converted. 

And, also for achieving the object mentioned above, 
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according to the present invent ion , there is further provided a 
program stored on a computer readable storage medium for assisting 
orthopedic surgical operation, comprising the followings steps 
of: a step for picking up an image of a surgical filed by means 
5 of an image pick-up apparatus ; a step for producing a stereographic 
image of the surgical filed, the image of which is picked up, in 
an image producing unit; a step for inputting reference points 
of a surgical operation route upon basis of a kind of the surgical 
operation and said stereographic image through an input unit; a 

10 step for calculating a smooth surgical operation route upon basis 
of said kind of the surgical operation inputted and the reference 
points in a surgical operation route calculation unit; a step for 
processing said stereographic image and said surgical operation 
route to be displayable in an image processing unit; and a step 

15 for displaying the images processed in said image processing unit 
on an image displaying apparatus . 

BRIEF DESCRIPTION OF THE SEVERAL. VIEWS OF THE DRAWING 

Those and other objects, features and advantages of the 
present invention will become more readily apparent from the 
20 following detailed description when taken in conjunction with the 
accompanying drawings, wherein: 

Fig . 1 is a block diagram of showing an embodiment of a surgical 
operation assistance system, according to the present invention; 
and 

25 Figs. 2 through 6 show each of the screens displayed on a 

display apparatus of the surgical operation assistance system shown 
in Fig. 1 mentioned above, wherein: 

Fig. 2 shows an example of the screen of the surgical field; 

Fig. 3 shows an example of a dialog screen of an arbitrary 
30 line input unit for use of showing a slice screen; 
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Fig. 4 shows an example of a dialog screen of a reference 
point input unit; 

Fig. 5 shows an example of a dialog screen of a positional 
information integration unit; and 

5 Fig. 6 shows an example of other dialog screen. 

DETAILED DESCRIPTION OF THE INVENTION 

Hereinafter, embodiments according to the present invention 
will be fully explained by referring to the attached drawings. 

Hereinafter, explanation will be given on a first embodiment 
10 of the present invention, by referring to Figs . 1 to 6. The present 
embodiment will be explained by an example, in particular, of being 
applied in a case, when conducting the surgical operation through 
the Rotating Acetabular Osteotomy (hereinafter, being called by 
RAO), with applying a surgical operation robot therein. 

15 First of all, explanation will be given by referring to Fig. 

2 attached herewith, in relation to the RAO. This Fig. 2 is a view 
for showing bones of a coax (i.e., a hip joint), and it is a view 
described in a document, "Surgical Exposures in Orthopedics", 
Stanley Md. Hopperfeld, et al . ; in particular, on page 344 of 

20 Japanese translation, edited by Terayama et al., (1998). 

In a case, when an acetabular roof 110 of the coax does not 
cover an upper part of a caput ossis femoris in the shape thereof, 
in a patient to whom the RAO is applied, due to hypoplasia, thereby 
causing a defective portion 111 in the condition thereof, for 
25 example, the body weight rests on it, under the condition that 
the caput ossis femoris 113 projects a little bit outside than 
the acetabular roof 110. For this reason, the force applying on 
the acetabular roof 110 rests , not being distributed over the entire 
surface thereof, but being concentrated at an edge thereof on a 
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central side of the body, therefore the patient feels the pains 
every time when she/he takes a step. 

The RAO is a such kind of surgical operation, that the 
acetabular roof 110 is cut off at the portion, which is surrounded 
5 by an osteotomy line (or bone cutting line) 112 shown by a thick 
line in the figure, thereby rotating the portion surrounded by 
the bone cutting line 112 in a direction of an arrow, so that an 
upper part of the caput ossis femoris 113 can be covered by the 
entire of the acetabular roof 110. 

10 In the present embodiment, it is assumed that the orthopedic 

surgical operation is done with using a surgical operation robot, 
so that the bone is cut smoothly into a spherical shape in accordance 
with the surgical operation plan made up , and that a cutting portion 
of the surgical operation is small in the range thereof, thereby 

15 enabling the patient to move independent early after having the 
surgical operation, with commonly using a brace outside a wound 
(i.e., an assisting tool for enabling the patient to walk before 
fixing the bones which are connected) , and it relates to a surgical 
operation assistance system being necessary in such the case 

20 mentioned above. 

Next , explanation will be explained on the surgical operation 
assistance system, according to the present embodiment, by 
referring to Fig. 1. This Fig. 1 is the structure view for showing 
the surgical operation assistance system. 

25 The surgical operation assistance system 4 comprises a 

surgical operation planning system 1, a surgical operation robot 
2, and a navigation system 3. This surgical operation system 4 
comprises a computer and various kinds of input devices, and also 
output devices, and so on. The surgical operation planning system 

30 1 comprises a surgical field shape grasp function unit 10 for 
grasping the condition of the surgical field, and a surgical 
operation route determine function unit 11 for determining the 
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surgical operation route, fitting to the condition of the surgical 
field. This surgical operation planning system 1 is constructed, 
so that it grasps the condition of the surgical field by means 
of the surgical field shape grasp function unit 10, and determines 
5 the suitable surgical operation route fitting to the determined 
condition of the surgical operation route, thereby checking the 
optimization of the surgical operation plan upon the basis thereof, 
in the surgical field shape grasp function unit 10. 

The computer comprises an image pick-up device 100, an image 
10 produce unit 101, an image extract unit 102, an image process unit 
103, an image display apparatus 104 for displaying the image thereon, 
an arbitrary line input unit 105 for use of showing a sliced image, 
a reference point input unit 108, a surgical operation route 
calculate unit 107, and a positional information integrate unit 
15 302. Those constituent elements are built up with programs, and 
also are executed in accordance with operation steps, which will 
be mentioned later. Further, the computer may be divided into a 
plural number of portions thereof. 

And, the surgical operation assistance system 4 is a system 
20 for making up a plan for the surgical operation with using the 
surgical operation planning system 1, before actually conducting 
the surgical operation, and it is that, during the surgical 
operation, for indicating or presenting the positional information 
of the surgical operation robot 2 with respect to the patient, 
25 to a surgical doctor, as well as, the surgical operation plan, 
laying it on the medical use image of the patient. In the present 
embodiment, the surgical operation planning system 1 and the 
navigation system 3 are built up, by using a portion thereof in 
common with. 

30 The surgical operation planning system 1 comprises the 

surgical field shape grasp function portion 10 for grasping the 
condition of the patient, and the surgical operation route 
determine function unit 11 for determining the surgical operation 
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route fitting to the condition of the surgical field. This surgical 
operation planning system 1 is constructed, so that the condition 
of the surgical field is grasped in the surgical field shape grasp 
function unit 10 , that the suitable surgical operation route is 
5 determined fitting to the condition of the surgical field in the 
surgical operation route determine function unit 11 , and upon the 
basis of those , the optimization of the surgical operation plan 
is checked, in the surgical field shape grasp function portion 
10. 

10 The surgical field shape grasp function portion 10 comprises 

the image pick-up device 100 for picking up an image of the surgical 
field, the image produce unit 101 for producing a slice image and/or 
a stereograph, and the image extract unit 102 for extracting an 
image of the necessary portion from the produced images, the image 

15 process unit 103 for processing the extracted image so that it 
can be displayed under a desired condition thereof, the image 
display apparatus 104 for displaying the image thereon, and the 
arbitrary line input unit 105 for use of the slice image, for 
designating a sliced image to be seen. 

20 This surgical field shape grasp function unit 10 picks up 

the medical use image in the vicinity of the surgical field through 
the image pick-up device 100, and reads a film of the medical use 
image, which is picked up, into the image produce unit 101, thereby 
producing the sliced image and/or the stereograph image thereof. 

25 And then, only the image of a portion of the bone is extracted 
from the stereograph produced in the image extract unit 102, and 
it is processed in the image process unit 103 , so that the stereograph 
of the bone extracted can be displayed, rotatably. Thus, in the 
structure mentioned above, this image is presented to the surgical 

30 doctor on the image display apparatus 104 . With this, the surgical 
doctor can check the shape of the bone , from various angles thereof . 

Further, the image pick-up device 100 is an instrument for 
picking up the image of the surgical filed, such as, an image of 
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an endoscope, an image of an electronic microscope, a picked-up 
image through a CCD camera, a MRI or CT image, an ultrasonic image, 
and others. The image display apparatus 104 is an instrument for 
displaying the image information, which is transmitted from the 
5 image process unit 103, on a CRT, a head mount display or other 
image display devices, or alternatively, that of projecting the 
image information on the patient, directly. 

In a case, when wishing to check the configuration of the 
bone on a sliced cross-section view thereof, so as to inspect the 

10 condition of the acetabular roof, in more detail, for example, 
in the planning of the surgical operation, the surgical doctor 
designates the slice image that she/he wishes to see through the 
arbitrary line input unit 105 for use of slice images. After being 
processed in the image process unit 103 , then the image is presented 

15 to the surgical doctor, in the form of the sliced image on the 
image display apparatus 104. This enables the surgical doctor to 
make up a suitable surgical operation plan. 

When designating the sliced image through the arbitrary line 
input unit 105 for the slice image, "Sliced Cross-Section Display 
20 by Arbitrary Line" is selected from a menu screen displayed on 
the image display apparatus 104 . With this , the screen of presenting 
the stereograph of the surgical filed appears on the image display 
apparatus 104, and also a dialog screen 123 appear, but at an end 
of that screen, as shown in Fig. 3. 

25 Then, when clicking at an arbitrary point on the stereograph 

of the surgical field through a mouse, the color of that point 
is changed, thereby designating a first passage point of the 
arbitrary line for use of the sliced image. However, when correcting 
that passage point, it is possible to alter the passage point to 

30 a new correction point, through clicking of the mouse at the 
correction point on the screen presenting the stereograph of the 
surgical field, after pushing down a correct button 120 at the 
point corresponding thereto on the dialog screen 123. When 
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designating a second passage point of the arbitrary line for use 
of the sliced image, after designating the first passage point, 
the mouse-click is made , but without pushing down the correct button 
120. With this, the point where the mouse-click is made is 
5 automatically acknowledged to be the second passage point. When 
designating the passage point, it is possible to designate it while 
rotating the stereograph of the surgical field, freely, through 
dragging of the mouse. 

When the second passage point is designated, a sliced image 
passing through the two (2) points, in the vertical direction, 
is displayed, in the place of the image presenting the stereograph 
thereof. If the direction of this sliced image is improper, the 
direction of this sliced image can be changed through moving it 
up to a desired sliced image, through dragging of the mouse. Herein, 
when pushing down a decide button 121 on the dialog screen 123, 
then the cross-section of the sliced screen is displayed thereon. 
This enables the surgical doctor to make up the suitable plan for 
the surgical operation. 

However, when pushing down a correct button 122 of the slice 
direction on the dialog screen 123, it is possible to designate 
the direction of this sliced image, again, starting from the timing 
of determining thereof. Also, when pushing down the correct button 
120 of the passage point 1 or 2 on the dialog screen 123, it is 
possible to designate that passage point, again, starting from 
the timing of determining thereof. 

The surgical operation route determine function unit 11 
comprises the reference point input unit 108 for inputting a 
reference of the surgical operation route, the surgical operation 
route calculate unit 107 for calculating out the surgical operation 
30 route upon the basis of the reference inputted, and a surgical 
operation route confirm unit , on which the surgical operation route 
calculated out is displayed. The surgical operation route confirm 
unit is built up by using the image process unit 103 and the image 
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display apparatus 104 in common with, as is shown in Fig. 1. 

The surgical operation route determine function unit 11 has 
the functions: such as, of calculating out a spherical surface 
passing through the reference point inputted in the surgical 
5 operation route calculate unit 107 , after the surgical doctor 
designates " Determine Ostectomy Line" among the kinds of surgical 
operations from the menu-bar displayed on the image display 
apparatus 104 and "Spherical Surface" among the kinds of surfaces 
and lines for approximating the surgical operation route, and also 

10 she/he inputs the reference points corresponding to that kind which 
is designated (e.g., four (4) points, in this case, because of 
the spherical surface) through the reference point input unit 108; 
and, of presenting an intersection line between the spherical 
surface calculated out and the bone on the image display apparatus 

15 104 of the surgical operation route confirm unit. However, the 
surgical operation route determine function unit 11 may describe 
a curved surface, smoothly, passing through the reference points 
inputted, like drawing software, through inputting the number of 
the reference points which she/he wishes to input from a list screen 

20 of the menu-bar, after the surgical doctor designates the 
"Determine Ostectomy Line" among the kinds of surgical operations 
from the menu-bar, thereby enabling to present it to the surgical 
doctor on the image display apparatus 104 of the surgical operation 
route confirm unit. 

2 5 When inputting the reference points through the reference 

point input unit 108, since a dialog screen 132, such as shown 
in Fig. 4 , appears on an end portion on the screen through inputting 
of the kind of surgical operation and the approximating surface 
of the surgical operation from the menu-bar; therefore, the first 

30 reference point is inputted with using this screen. The inputting 
of the reference point via the reference point input unit 108 is 
carried out, through dragging of the mouse, so as to freely rotate 
the stereographic image displaying an arbitrary point to be 
inputted as the reference point, and also clicking of the mouse 
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at that arbitrary point. In this instance , the point at which the 
input is made changes in the color thereof , therefore it is possible 
to confirm that the input is made, easily. However, when correction 
is made on the reference point, after pushing down the correct 
5 button 130 at the point corresponding thereto on the dialog screen 
132, the clicking of the mouse is made at the reference point on 
the stereographic image, again, thereby achieving the change of 
the reference point to a new correction point. When designating 
the second point and also others following thereafter, after 
10 designating the last reference point (i.e., the first one), the 
mouse is clicked, but without pushing down the correct button 13 0. 
With this , the point at which the mouse is clicked is automatically 
acknowledged to be the next passing point. 

When a decide button 131 is pushed down on the dialog screen 
15 shown in Fig. 4 by the surgical doctor, after completing the inputs 
of all the four (4) reference points, the Osteotomy Line is 
calculated out in the surgical operation route calculate unit 107 . 
The surgical operation route calculating unit 107 calculates out 
an equation of the spherical surface in the coordinate system of 
20 the medical use image for presentation, passing through the all 
of the four (4) reference points, and it further determines on 
whether the numerical values of the medical use image data on the 
equation of the spherical surface represents the bone or not, 
thereby calculating out the Ostectomy Line on the surface where 
25 the equation of the spherical surface comes crossing the bone. 
After being converted into the image information in the image 
process unit 103, the Ostectomy Line is presented to the surgical 
doctor on the image display device 104. 

Study of the Ostectomy Line is conducted while simulating 
30 the surgical operation in the image process unit 103; e.g., by 
rotating the bone surrounded by the Ostectomy Lines through 
dragging of the mouse with pushing down the other button, being 
opposite to that used for rotation of the entire stereographic 
image, in the horizontal direction thereof. In that instance, the 
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surgical doctor can make confirmation at an arbitrary timing on 
a relationship between the Osteotomy Lines and an important organ 
therearound; e.g. , on whether the important organ would be injured 
or not when rotating the bone, or on whether it is difficult or 
5 not when cutting open the tissues up to the osseous tissue after 
cutting open the skin, etc. , by extracting the image of the important 
organ, such as, an tendon in the vicinity of the bone of the surgical 
field, etc., for example, within the image extract unit 102, and 
displaying it /them put on the image of the bone. Also, in the image 
10 process unit 103, it is possible to make confirmation on a 
relationship between the Ostectomy Lines and the important organ ( s ) , 
while deleting or moving the tissues in a front of the Ostectomy 
Lines within the image extract unit 102. 

The correction of the Ostectomy Line can be done by clicking 
15 the mouse at the reference point on the stereographical view, while 
pushing down the button at the reference point on the dialog screen 
shown in Fig. 4 , on which the correction wished to be made, thereby 
clicking the mouse at the reference points on the stereographical 
image, again. After completing the correction, when the decide 
20 button 131 is pushed down on the dialog screen 132 shown in Fig. 
4 by the surgical doctor, the Ostectomy Line(s) is /are presented 
to the surgical doctor, on which the spherical surface passing 
through the four (4) reference points comes crossing with the bone. 

The navigation system 3 comprises: a marker 300; a 
25 three-dimensional position measurement apparatus 301; a position 
information integrate unit 302; an image pick-up device 100; an 
image produce unit 101 ; an image extract unit 102 ; an image process 
unit 103; and an image display apparatus 104. However, the image 
pick-up device 100, the image extract unit 102, the image process 
30 unit 103 and the image display apparatus 104 are used in common 
with the surgical field shape grasp function unit 10, as shown 
in Fig. 1. 



The navigation system 3 obtains a conversion matrix for 
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converting the position coordinate system of an actual patient 
into the position coordinate system of the medical use image of 
the patient, which was picked up before making up the surgical 
operation plan, in the position information integrate unit 302 
5 thereof , and it multiplies the position coordinates of the surgical 
operation tool onto the coordinate system of the actual patient 
with that conversion matrix obtained, thereby obtaining the 
coordinates of the surgical operation tool on the coordinate system 
of the medical use image of the patient. Thereafter, it produces 

10 an image for presenting the surgical operation tool on the medical 
use image of the patient, which is picked up before making up the 
surgical operation plan, in the image produce unit 101, and it 
produces an image, laying that image on the surgical operation 
route, which was produced in the surgical operation route calculate 

15 unit 107, within the image process unit 103, thereby presenting 
them to the surgical doctor on the image display apparatus 104, 
on a real-time base. 

The surgical operation robot 2 comprises : a manipulator (i.e., 
a robot arm) 201 for holding the surgical operation tool thereon; 

20 and a manipulator controller apparatus 200 for operating it. The 
operation of the surgical operation robot 2 can be selected to 
operate, automatically or manually. When operating the surgical 
operation robot 2, automatically, the result calculated out in 
the surgical operation route calculate unit 107 is transmitted 

25 to the manipulator controller apparatus 200, so as to perform the 
surgical operation while controlling the manipulator 201 in 
accordance with the surgical operation route planned. 

When operating the surgical operation robot 2, manually, 
the surgical operation is conducted through operation of the 
30 manipulator 2 01 while transmitting the control information from 
an operation table to the manipulator controller apparatus 2 00. 
However, when performing the surgical operation, manually, it is 
conducted while confirming the shifting between the surgical 
operation plan and the actual surgical operation, with using the 
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navigation system 3 . 

Herein, to be the medical use image of the patient picked 
up before making up the plan of surgical operation , the medical 
use image is used , which was used in the surgical operation planning 
5 system 1. However, the image was picked up under the condition 
that at least three (3) markers or more are attached thereupon, 
when being picked up at first by means of the image pick-up device 
100. The conversion matrix, for converting from the position 
coordinate system of the actual patient into the position 

10 coordinate system on the medical use image of the patient, which 
was picked up before making up the surgical operation plan, can 
be obtained by the following manner. Thus, first of all, the same 
number of the markers 300 are attached on the actual patient at 
the positions where the markers of a number of at least three (3) 

15 or more are attached, when picking up the medical use image of 
the patient before making up the surgical operation plan, and the 
position coordinates of those markers are measured through the 
three-dimensional position measurement apparatus 301, and thereby 

indicating them by means of a matrix of 3x3, or more than that. 

20 A matrix, for converting that matrix into the matrix of 3x3 or 
more than that for representing the marker position coordinates 
on the medical use image of the patient, which was picked up before 
making up the surgical operation plan, is obtained through 
calculation conducted within the position information integrate 

25 unit 302. 

The position coordinates of the surgical operation tool can 
be obtained on the position coordinate system of the actual patient , 
as will be mentioned below. First of all, the three (3) reference 
pints of the surgical operation robot 2 are measured by means of 
30 the three-dimensional position measurement apparatus 301 on the 
position coordinate system of the actual patient. Next, in the 
position information integrate unit 302 , from the designed numeral 
values of the surgical operation robot 2 and the reference position 
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of the surgical operation robot 2 that is obtained from the control 
information outputted from the manipulator controller apparatus 
200 , the conversion matrix up to the surgical operation tool held 
at a tip of the manipulator is multiplied onto the coordinates 
5 of the three (3) reference points of the surgical operation robot 
2 in the position coordinate system of the actual patient. With 
this, the position coordinates of the surgical operation tool 
mentioned above can be obtained. 

In the position information integrate unit 302, inputs of 
10 the positions of the markers on the medical use image and the 
positions of the markers 300 attached on the actual patient, as 
well as, inputs of the reference points of the surgical operation 
robot, are conducted by the following manner. 

When selecting "Navigation Initial Setting" from the 
15 menu-bar on the image display apparatus 104, the dialog screen 
314, such as shown in Fig. 5, appears at an end portion of the 
screen . The surgical doctor conducts the inputting after completing 
selection of one to be inputted, from a tab "Medical Use Image" 
310, a tab "Actual Patient" 311, and a tab "Surgical Operation 
20 Robot" 312, which are displayed on the dialog screen 314. 

Herein, when inputting the position of the markers on the 
medical use image, after selection of the tab "Image For Medical 
Use" 310 in Fig. 5, the stereographic image is rotated freely on 
the screen of the medical use image through dragging the mouse, 

2 5 thereby displaying an arbitrary point , at which an input is desired 
to be made, as an input point. Under this condition, a first one 
is inputted through clicking of the mouse at the arbitrary point. 
When correcting, it is achieved by clicking the mouse again, after 
pushing down the correct button 313 at the point corresponding 

30 thereto on the dialog screen 314 shown in Fig. 5. In a case of 
designating a second one and also others thereafter, the point 
where the mouse is clicked is automatically acknowledged to be 
a next input point, when clicking the mouse, but without pushing 
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down the correction button after designating the last input point. 

When inputting the positions of the markers 3 00 on the actual 
patient, after selecting the tab " Actual Patient" 311 in Fig. 5, 
the position information of the markers are transmitted to the 
5 position information integrate unit 302 , while putting a measuring 
probe of the three-dimensional position measurement apparatus 301 
on the markers 300 attached on the patient, in the same order or 
sequence when inputting the positions of the markers on the medical 
use image. When completing the transmission thereof, since the 

10 position coordinate numerical values are displayed, such as, like 
the screen shown in Fig . 6 , an input button 3 1 5 of the corresponding 
marker is pushed down after confirming the display thereof. When 
designating a second input point and also others thereafter, after 
designating the last input point, the measuring probe of the 

15 three-dimensional position measurement apparatus 301 is also put 
on the marker 300 attached on the patient, but without pushing 
down a correct button 316. With this, the position information 
of the marker 300 is transmitted to the position information 
integrate unit 3 02, and the position coordinate numerical values 

20 314 are displayed , in the s imilar manner when inputting the pos it ion 
information of the first marker, therefore the input button 15 
can be pushed down after confirming the display thereof. However, 
when correcting, it can be made by putting the measuring probe 
of the three-dimensional position measurement apparatus 301 on 

25 the markers attached on the patient, again, after pushing down 
the correct button at the corresponding point on the dialog screen 
shown in Fig. 6. 

When inputting the reference points of the surgical operation 
robot, the operation is conducted, as will be mentioned in the 
30 following. When selecting the tab "Surgical Operation Robot" 312 
on the dialog screen shown in Fig. 5, a maker position input screen 
for the surgical operation robot is displayed, in the s imilar manner 
to the marker position input screen of the actual patient shown 
in Fig. 6. On that screen, the measuring probe of the 
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three-dimensional position measurement apparatus 301 is put on 
the reference points, in accordance with a predetermined order 
or sequence thereof. With this, the position information of the 
reference points are transmitted to the position information 
5 integrate unit 3 02, and the position coordinate numerical values 
314 are displayed, in the similar manner to the dialog screen shown 
in Fig. 6, therefore the input button 315 can be pushed down after 
confirming the display thereof. However, when making correction, 
it is only possible by putting the measuring probe of the 
10 three-dimensional position measurement apparatus 301 on the 
reference points , again, but after pushing down the correct button 
316 at the corresponding point on the dialog screen, which is same 
to that when inputting. 

Since the dialog screen is displayed for the purpose of 
15 confirming the completion of all the inputs, when inputting all 
of the position coordinates of theall points , suchas, the positions 
of the markers on the medical use image, the position of the markers 
attached on the actual patient, and the reference points of the 
surgical operation robot, etc., then the input can be completed 
20 by selecting "OK" on this dialog screen. 

The screen, for presenting the position of the surgical 
operation tool on the medical use image of the patient that was 
picked up before making up the surgical operation plan, is 
selectable from among a plural number of the screens . Namely , either 

25 one can be selected by the surgical doctor: e.g., a screen of 
displaying 3-D images of the surgical operation robot and the 
surgical operation tool, being piled up on the medical use image 
of the patient that was picked up before making up the surgical 
operation plan; or, a screen displaying 3-D image of only the 

30 surgical operation tool, being piled up on the medical use image 
of the patient that was picked up before making up the surgical 
operation plan. In a method for that selection, selection is made 
on either "Robot and Tool" or "Only Tool" from the menu-bar, after 
selecting "Type Selection of Navigation Image". However, this 
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selection can be also made during conduction of the surgical 
operation. And also, the position coordinates of the surgical 
operation tool within the position coordinate system of the actual 
patient may be obtained, through measuring the positions of the 
5 markers, which are directly attached on the surgical operation 
tool, by using the three-dimensional position measurement 
apparatus 301. 

As a method for presenting the position of the surgical 
operation tool(s) on the medical use image of the patient, which 

10 was picked up before making up the surgical operation plan, the 
images of all the surgical operation tools to be used are picked 
up, before conducting the surgical operation by means of the image 
pick-up device 100, and this 3-D image obtained is displayed, while 
piling it up on the medial use image of the patient. However, the 

15 3D image of the surgical operation robot is inputted into a film, 
which was taken in advance. As a method for selecting the image 
of the surgical operation tool, every time when changing the 
surgical operation tool, " Selection of Tool File" is selected from 
the menu-bar, and then the tool file is selected from a dialog 

20 screen for opening the file. 

According to the present embodiment , in the stage of planning 
the surgical operation, it is possible to provide the surgical 
operation assistance system, enabling to draw the smooth surgical 
operation route, being necessary , in particular , in the osteotomy , 

25 such as, the RAO with an aid of the surgical operation robot, etc. , 
for example. Also, since the surgical operation route can be 
determined to be the smooth surface, but not aggregation of lines, 
therefore input items, being necessary when determining the smooth 
surgical operation route, come to be less in the number thereof 

30 than that of a method of determining inflection points on the way 
thereof, thereby obtaining elimination or reduction of labors for 
the surgical doctor, and further, it is possible to provide the 
surgical operation assistance system being applicable into 
orthopedic surgery, in general. 
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Next, explanation will be given on a second embodiment 
according to the present invention, which is applied when 
conducting the surgical operation other than that mentioned above, 
i.e. , by means of the RAO with using the surgical operation robot. 

5 In this second embodiment, a reference point input unit is 

provided for the purpose of inputting the kind of the surgical 
operation, such as, the osteotomy or drilling, for example, in 
an input format of a button, a slider, a menu-bar, or others, 
displayed on the screen; or, for the purpose of inputting the kind 

10 of lines for use of the approximation, in an input format of the 
button, the slider, the menu-bar or others , displayed on the screen, 
in particular, on the surgical operation, in which a portion of 
the surgical operation route can be approximated by an ellipse, 
acylindroid, a circle, a column, a parabola, a parabolic cylinder, 

15 a straight line, a rectangular, a parallelopipedon, or a sphere. 
Also, the 3-D position coordinates of the number of reference points , 
being necessary for corresponding to the kind of lines for the 
approximation, can be designated by clicking of the mouse or a 
touch panel, on the stereographic medical use image of the patient, 

20 which was picked up before the surgical operation. 

As is apparent from the explanation given on various 
embodiment in the above, according to the present invention, it 
is possible to determine the smooth surgical operation route to 
the surgical field, easily, thereby achieving the surgical 
25 operation assistance system and the surgical operation assisting 
method, and also software thereof, enabling easy orthopedic 
surgical operation, with certainty. 

The present invention may be embodied in other specific forms 
without departing from the spirit or essential feature or 
30 characteristics thereof. The present embodiment (s ) is /are 
therefore to be considered in all respects as illustrative and 
not restrictive, the scope of the invention being indicated by 
the appended claims rather than by the forgoing description and 
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range of equivalency of the claims are therefore to be embraces 
therein. 



